When playing back a video from a storage medium to a display, such as a video stored on a digital versatile disc (DVD) played on a television screen, the frame rate of the display may be greater than that of the source material. Many modern televisions allow for an increased frame rate, which may for example be 100 Hz, whereas the source material may have a frame rate of 50 Hz or less. This higher frame rate can reduce the flicker that may otherwise be apparent to a user.
One technique for up-scaling the frame rate, for example from 50 Hz to 100 Hz, would be simply to repeat input frames in the output video signal. This technique, however, does not smooth out motion in the video and can result in visual artifacts becoming apparent as jerkiness in the up-scaled display.
Motion compensated frame rate up-conversion techniques are therefore typically used to increase frame rates, while attempting to reduce such artifacts. Such motion compensation involves intermediate frames being generated by interpolation between successive input frames. By estimating motion vectors for blocks in successive input frames, portions of the input frames that correspond to moving objects can be translated to intermediate positions in the interpolated frames. This results in motion being apparently smoothed out in the output video sequence.
The quality of interpolated images in picture rate up-conversion is predominantly dependent on the accuracy of the motion vector fields. Block-based motion estimation techniques typically result in incorrect vectors in occlusion regions, which leads to so-called halo effects in an up-converted video sequence. This can occur when a foreground object in the video sequence is passing in front of a background object. Either one or both of the foreground and background objects may be moving relative to the boundary of the video frames. In the case of a static camera filming a moving object, the foreground object moves against a static background. In the alternative, a moving foreground object may be tracked, resulting in an apparently moving background. Combinations of these situations are also common.
When a foreground object changes position from a first frame to a second frame, an area in front of the object, that is to say immediately adjacent the edge of the object (to the right of the object, if moving from left to right) is occluded in the second frame, and a further area behind the object (to the left of the object, if moving from left to right) is revealed in the second frame. Such areas are known as occlusion areas or regions, and are characterized as uncovering occlusion regions and covering occlusion regions. By definition, uncovering occlusion regions are those parts of a frame that were not present in a previous frame, typically the immediately preceding frame, while covering occlusion regions are those parts of a frame that are not present in a subsequent frame.
The presence of occlusion regions results in common techniques for motion compensated frame rate conversion creating artifacts in the output up-scaled video known as halo effects. A halo effect is caused when an interpolated frame displays a portion of an occlusion region incorrectly, due to the inability of the conversion technique to determine the correct image data to use in the case of an interpolated occlusion area. Various different techniques have been proposed to solve this occlusion problem, with varying degrees of success, as for example disclosed in the references (i) M. Mertens and G. de Haan, entitled “Motion Vector Field Improvement for Picture Rate Conversion with Reduced Halo”, Proc. of the SPIE/IST VCIP, San Jose, 21-26 Jan. 2001, pp. 352-362 (ii) R. B. Wittebrood, G. de Haan and R. Lodder, entitled “Tackling Occlusion in Scan Rate Conversion Systems”, Digest of the ICCE'03, June 2003, pp. 344-45 and (iii) WO 96/28931 entitled “Motion-compensated Interpolation”. However, none of the proposed techniques achieve the desired combination of a large reduction of halo artifacts together with an adequate robustness to the different types of input material that might be encountered.
In the publication by E. B. Bellers, J. W. van Gurp, J. G. W. M. Janssen, R. Braspenning, R. Witterbrood, entitled “Solving Occlusion in Frame Rate Up Conversion”, Digest of International Conference on Consumer Electronics, January 2007, pp. 1-2, a further technique is disclosed dealing with occlusion regions. In general terms, this technique comprises five main steps:                1: Estimate, for the previous and current video frames, the forward and backward motion vectors;        2. Combine each pair of motion fields to form an occlusion-free motion vector field;        3. Generate an interpolated ('retimed') vector field in-between the two combined vector fields;        4. Determine the occlusion area and occlusion type (i.e., covering or uncovering); and        5. Perform interpolation according to retimed motion vector field and occlusion mask.        
WO 2005/027525 entitled “Motion Vector Field Re-timing” and WO 2005/022922 entitled “Temporal Interpolation of a Pixel on Basis of Occlusion Detection” disclose methods of performing steps 3 and 4. In all known techniques, detection of occlusion in video frames is an essential step. This step can be performed implicitly or explicitly. In the publication by R. B. Wittebrood et al identified above, a implicit detection technique is proposed, comprising the following steps:                1. Find the occlusion area;        2. Determine the occlusion type;        3. Calculate the background motion vector; and        4. Perform a robust up-conversion using the occlusion type and background motion vector.The techniques disclosed in the publication by R. B. Wittebrood et al, as well as in references WO 2005/027525, WO 2005/022922 and WO 2007/063465 entitled “Motion Vector Field Correction” seek to determine a mismatch between estimated motion vectors to detect occlusion, and/or to relate the magnitude of a match error (sometimes called a residue) to an amount of occlusion.        
Such known methods of explicit occlusion detection are, however, not ideal. Determining a mismatch suffers from the frequent occurrence of poor motion vectors, whereas relating the magnitude of a match error to an amount of occlusion suffers from the fact that the absolute match error relates poorly to the presence of occlusion. For example, a good match in high contrast areas might provide a higher match error than a poor match in occluded areas with low contrast. Failure in occlusion detection frequently means subsequent steps in the algorithm will fail, thereby leading to visually annoying artifacts in the resulting interpolated image.
The listing or discussion of a prior-published document in this specification should not necessarily be taken as an acknowledgement that the document is part of the state of the art or is common general knowledge.
There is a need for an improved method of occlusion area detection and identification, in particular to improve the robustness of such detection and identification.